Skip to content

Agents: the workflow tool sends then attaches, so a crash no longer reports a run as failed; the CLI shows steered messages - #306

Merged
igalshilman merged 7 commits into
restatedev:mainfrom
igalshilman:claude/workflow-rejoin-steered-cli
Nov 9, 2026
Merged

igalshilman merged 7 commits into
restatedev:mainfrom
igalshilman:claude/workflow-rejoin-steered-cli

Conversation

@igalshilman

@igalshilman igalshilman commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Two fixes found while sanity-checking modal-template against 1.5.0. Python only; no change to the native extension.

1. The workflow tool sends to the workflow, then attaches to the send's invocation

Agents are session objects. After a crash they run again from their first line, so a tool call cut off by the crash is made again. The agents' Restate tools rely on idempotency keys for that: a repeated call is the same call. A workflow's main handler is the exception. Restate takes no idempotency key for it, and refuses a second call with 409, "the workflow method was already invoked".

Before (1.4.2 and 1.5.0): if the process died while a workflow tool's workflow ran, the retried turn told the model Error: the workflow method was already invoked, even though the workflow ran to the end. I reproduced it with modal-template's support example: kill the process during the refund. The turn records the error, while sys_invocation shows refund/…/run completed successfully. LangGraph's nodes.workflow has the same problem.

Now: the workflow tool and nodes.workflow always send to the workflow, take the invocation id from the send, and attach to that invocation for the result (run_workflow).

  • A workflow runs once per id. A send to one whose run already started starts nothing, and its invocation id is that run's. So a tool call made again after a crash, or a key that ran before, waits for that run.
  • No call is refused and no error text is matched.
  • The run is the workflow's own, not a call of the turn, so stopping the turn doesn't cancel it.
  • restate.sessions.internal gains start (a send made right away that returns its invocation id) and attach_invocation (over the SDK's attach_invocation), for libraries.
  • restate.sessions' ctx.call is unchanged from 1.5.0.

2. The agents CLI shows the user's messages it didn't type

Before: after the conversation so far, the terminal hid every user message, taking them as typed here. A /steer message never showed where the turn took it, and messages from other clients didn't show at all.

Now: it notes the messages it sends to chat, and hides only those, once each by text. That includes lines typed before the first change arrived, which pullfrog pointed out. Every other user message shows in its place:

  ← approved
you: And where is order 104?
  → refund({"order": "302", "amount": 25.0})

Tests

Against a real server (harness):

  • servercontext.py: a workflow returns "hi". A service sends to it twice: both sends give the same invocation id, and attaching to it returns "hi".
  • agents_tools_server.py (new): the workflow tool runs again after a crash that followed its send, and answers "hi". With 1.5.0's call instead, it answers Error: the workflow method was already invoked.

In-process (fake host):

  • The tool sends and attaches, again under the same id, and makes no call.
  • sessions.internal.start and attach_invocation.
  • agents_cli.py (new): what the terminal shows for typed, steered and others' messages.

Runs:

  • just test-fast: 358 passed.
  • Against a local Restate 1.8.1: agents_tools_server, agents_langgraph_nodes, sessions_harness and servercontext pass. On an earlier head, the full server suite gave 256 passed and 3 failed. Those 3 register host.docker.internal, which doesn't resolve without Docker.
  • ruff, pyright and mypy are clean.

🤖 Generated with Claude Code

… run it started

A session object runs again from its first line after a crash, so it calls
again what the crashed attempt called. An idempotency key makes that the same
call, except for a workflow's main handler: Restate takes no idempotency key
for it, and refuses the second call (409, "the workflow method was already
invoked"). The agents' `workflow` tool, and LangGraph's `nodes.workflow`, then
told the model that the workflow failed, while it ran to the end.

A call to a workflow's main handler (`ctx.call(wf.main, ..., key=id)`) that is
refused this way now attaches to the workflow's run by its id, through the
shared core's `AttachInvocationTarget::WorkflowId` (`attach_workflow` on the
VM and the server context), and returns its result: the run of that id,
whoever started it. Any other 409 (a cancellation), a workflow's shared
handlers, and calls by name are unchanged.

Tests: the rejoin and what isn't rejoined (fake host), the agents' workflow
tool after a crash, and a session handler that crashes once its workflow call
is made, against a real server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…here the turn took it

After the conversation so far, the terminal showed none of the user's
messages, since it took them to be typed here. So a message sent with
`/steer` never showed where the turn took it (between a tool's result and the
next step), and messages from other clients didn't show at all.

The terminal now notes the messages it sends to `chat`, and hides only those
(once each, by text); every other message of the user shows in its place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues. One small edge case in the CLI, noted inline.

Reviewed changes

I reviewed both commits: a session call made again to a workflow's main handler now attaches to the run instead of failing with 409, and the agents CLI now shows user messages that weren't typed in this terminal.

  • attach_workflow binding: PyVM.attach_workflow wraps shared-core's AttachInvocationTarget::WorkflowId with scope: None. The field matches the vendored 8.4 crate. VMWrapper, ServerInvocationContext and the session host pass it through.
  • Rejoin in CallFuture: ctx.call on a handler with kind == "workflow" gets a rejoin. If the call fails with a 409 whose message contains "already invoked", it awaits the attach. Cancellation 409s, shared handlers and generic_call by name keep their current behaviour.
  • Agents/LangGraph docstrings: these now say a workflow runs once per id, and that a repeated key gets the earlier run's result.
  • CLI Terminal: texts sent to chat are recorded with sending/unsent. _typed hides one user message per recorded text and shows every other user message, so steered messages and messages from other clients now appear.
  • Tests: the fake host gains refusals and attached; there is an in-process rejoin test, an agents tool test, a Docker harness test on both stores, and a new tests/agents_cli.py.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

Comment thread python/restate/agents/__main__.py Resolved
A harness test of the new API: a workflow returns "hi", a service sends it a
run (ignoring the result), attaches to it with `attach_workflow`, and answers
with what the attach returned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…up its text there

The first change (the conversation so far) shows every message, so it never
used up the text of a line typed before it came. A later user message of the
same text (steered, or from another client) was then hidden in its place.
The first change now uses it up too, while still showing the message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found. The CLI edge case from the last review is fixed.

Reviewed changes

Since the last review, one commit changed the agents CLI and added a test for it.

  • History change uses up typed text: Terminal.show now calls _typed for every user message, including those in the first change (the conversation so far). It still shows them there. So text typed before that change arrives no longer stays in self.sent and hides a later message with the same text, such as a /steer message or one from another client.
  • Test: tests/agents_cli.py adds a case that types Hi before the history arrives, then checks that a later Hi from another client is shown.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

…agents' tools rather than in ctx.call

The rejoin moves up, out of restate.sessions' `ctx.call`, which is back to
what 1.5.0 does: a call to a workflow's main handler that Restate refuses is
that error again, for every session object.

The agents' `workflow` tool and LangGraph's `nodes.workflow` now run the
workflow through `run_workflow`: they call its main handler, and if Restate
refuses the call because the run of that id started (409, "already invoked":
the tool call run again after a crash), they wait for that run with
`attach_workflow` and return its result. A first call stays a call of the
attempt, so stopping the turn still cancels it.

`restate.sessions.internal.attach_workflow(ctx, name, key)` is how a library
reaches the server context's `attach_workflow` from a session object's
context.

Tests: the helper (fake host); the workflow tool rejoins on "already invoked"
and not on another 409; against a real server, a workflow tool call run again
after a crash that followed its call answers with the run's result (it
answers "Error: the workflow method was already invoked" without the rejoin).
The sessions-level rejoin tests are gone with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@igalshilman igalshilman changed the title Sessions: a workflow call made again after a crash waits for its run; the agents CLI shows steered messages Agents: a workflow tool call run again after a crash waits for its run (attach_workflow); the CLI shows steered messages Nov 9, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found. The workflow rejoin now lives in the agents' tools, and restate.sessions' ctx.call is back to its 1.5.0 behaviour.

Reviewed changes

Since the last review, one commit moved the rejoin for "already invoked" workflow calls out of restate.sessions and into the agents' tools.

  • Reverted sessions call rejoin: _calls.py no longer has rejoin or already_invoked. _target and CallFuture are back to their original form, and the _api.py docstring and the sessions_harness.py rejoin test are gone. CallHost keeps attach_workflow, now typed Awaitable[bytes].
  • Added sessions.internal.attach_workflow: it accepts only an attempt or view context, raises TypeError for any other context, and delegates to the call host.
  • Added run_workflow in agents/_tools.py: it calls the main handler as before. When Restate refuses the call with a 409 whose message says "already invoked", it attaches to the run by its workflow id and deserializes the result with the handler's output serde. The workflow tool and LangGraph's nodes.workflow both use it. Any other 409 is raised again unchanged.
  • Tests: tests/agents_tools.py covers both the rejoin and the non-rejoin 409 case. The new tests/agents_tools_server.py covers a crash after the run started, against a real Restate. In sessions_object.py, the rejoin tests are replaced by a test of internal.attach_workflow. Locally, agents_tools.py and sessions_object.py pass (34 tests) and ruff is clean. The Docker-backed tests can't run on this runner.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

… send's invocation

A workflow runs once per id, and Restate takes no idempotency key for its main
handler. The agents' `workflow` tool, and LangGraph's `nodes.workflow`, now
always send to the workflow, keep the send's invocation id, and attach to that
invocation for the result (`run_workflow`). A send to a workflow whose run
started already starts nothing, and its invocation id is that run's: a tool
call made again after a crash, or a `key` that ran before, waits for that run.
No call is refused, and no error is matched. The run is the workflow's own:
stopping the turn doesn't cancel it.

`restate.sessions.internal` gains `start` (a send right away that returns its
invocation id) and `attach_invocation`, for libraries. The session-context
`attach_workflow` helper of the previous commit is gone; the SDK's
`attach_workflow` stays.

Tests: against a real server, two sends to one workflow id give the same
invocation id, which attach returns "hi" for; and the workflow tool run again
after a crash that followed its send answers "hi". The tool and the helpers,
against the fake host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@igalshilman igalshilman changed the title Agents: a workflow tool call run again after a crash waits for its run (attach_workflow); the CLI shows steered messages Agents: the workflow tool sends then attaches, so a crash no longer reports a run as failed; attach_workflow; the CLI shows steered messages Nov 9, 2026
… tool attaches to

The workflow tool gets the run's invocation id from its send, and attaches to
it with `attach_invocation`, so attaching by workflow id is not needed: the
binding (`PyVM.attach_workflow`), `VMWrapper.attach_workflow`,
`ServerInvocationContext.attach_workflow` and its harness test go. This PR
changes no Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@igalshilman igalshilman changed the title Agents: the workflow tool sends then attaches, so a crash no longer reports a run as failed; attach_workflow; the CLI shows steered messages Agents: the workflow tool sends then attaches, so a crash no longer reports a run as failed; the CLI shows steered messages Nov 9, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found. The workflow tool now sends, then attaches to the send's invocation, so nothing is matched on error text and the native attach_workflow binding is gone.

Reviewed changes

Since the last review, two commits replaced the "call, then attach on 409" approach. The workflow tool now always sends to the workflow and attaches to the invocation id of that send.

  • Removed attach_workflow: src/lib.rs, vm.py, server_context.py, _ContextHost and internal no longer have it. Without it, the PR doesn't touch the native extension.
  • Added internal.start and internal.attach_invocation: start sends right away with the new send_to helper and returns the send's invocation id. attach_invocation waits through the SDK's existing attach_invocation. Both accept only an attempt context.
  • Simplified run_workflow: it is now start followed by attach_invocation, and the result is deserialized with the handler's output serde. already_invoked and the idempotency key parameter are gone. Both the workflow tool and LangGraph's nodes.workflow use it. Stopping a turn still cuts the wait short, because Stop.guard cancels the framework task that awaits the attach. Only the workflow run itself keeps going, which the docstring says.
  • Updated tests: the fake host's generic_send now returns a handle whose invocation id is derived from the target. The tool test runs the tool twice and checks that this gives two sends, two attaches to the same id, and no calls. The server tests check that a second send gives the same invocation id and that the tool returns the result after a crash. Locally, agents_tools.py, sessions_object.py and agents_cli.py pass (37 tests), ruff is clean, and pyright reports no errors on sessions/ and _tools.py. The Docker-backed tests can't run on this runner.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@igalshilman
igalshilman merged commit 8339bbe into restatedev:main Nov 9, 2026
9 checks passed
@igalshilman
igalshilman deleted the claude/workflow-rejoin-steered-cli branch November 9, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants